Meta-instrumentation for security analysis

ABSTRACT

A system and method for analyzing and/or testing member devices in a multi-device system. The multi-device system includes a device-under-analysis (DUA) and a device-under-observation (DUO). An analyzer that is external to the multi-device system generates and sends test messages to the DUA. The analyzer monitors the health of the multi-device system through the DUO and detects a system-wide impact of the DUA caused by the test messages. The analyzer analyzes the DUA based on the test messages and the system-wide impact.

BACKGROUND

The present invention relates to automated analysis and testing ofhardware and/or software.

A software application or hardware device (collectively called a targetdevice or a device-under-analysis) can be analyzed or tested in anautomated way by using a second device called an analyzer. In thissituation, the analyzer generates a test message (e.g., an invalid testmessage), delivers the test message to the target device, monitors thetarget device, and/or analyzes the monitored information to determinewhether the target device is operating correctly. Analyses and testmessages identify the limitations of a target device.

The analyzer can determine whether a target device was designed andimplemented correctly by delivering various test messages to the targetdevice and observing and analyzing its responses to the tests. Forexample, security analysis can be performed as described in U.S. Utilitypatent application Ser. No. 11/351,403, filed on Feb. 10, 2006, thedisclosure of which is hereby incorporated by reference in its entirety.The various tests may include valid test messages (calledinstrumentation vectors) to determine whether the device is stillresponding appropriately. For example, the analyzer can identify andcharacterize failures in a target device based on its responses toinstrumentation vectors as described in U.S. Utility patent applicationSer. No. 11/760,600, filed on Jun. 8, 2007, the disclosure of which ishereby incorporated by reference in its entirety.

Observing a target device's responses to tests and instrumentationvectors may be insufficient in assessing (or analyzing) the targetdevice's design and/or implementation. The tests may cause the targetdevice to send invalid (or improper) messages to other devices, evenwhile the target device is still responding properly to the invalid testmessages and instrumentation vectors. This is especially problematicwhen the target device is a part of a larger system of multiple devicesbecause while the target device itself may not fail, it may neverthelessnegatively affect the health of the system in the course of processingthe tests.

For example, in a network using Open Shortest Path First (OSPF) routingprotocol, a router under attack may improperly communicate with otherrouters and corrupt their routing tables, link state databases (LSDBs),and/or other shared resources. The router itself may still respond toattacks correctly. However, the other routers with contaminated routingtables, LSDBs, and/or other shared resources may fail or malfunction asa result of the attacks. As another example, an enterprise system mayinclude a web server hosting web applications connected with a databaseserver. Attacks sent to the web applications may corrupt data stored inthe database server, even though the web applications may still appearnormal.

From the above, there is a need for a system and method to test andanalyze a target device to ensure that invalid traffic would not causethe target device to negatively affect the health of a system of whichthe target device is a member.

SUMMARY

The present invention provides a system and method for analyzing and/ortesting member devices in a multi-device system. The multi-device systemincludes a device-under-analysis (DUA) and a device-under-observation(DUO). An analyzer that is external to the multi-device system generatesand sends test messages to the DUA. The analyzer monitors the health ofthe multi-device system through the DUO and detects a system-wide impactof the DUA caused by the test messages. The analyzer analyzes the DUAbased on the test messages and the system-wide impact.

Other aspects of the disclosure include software, systems, components,and methods corresponding to the above, and applications of the abovefor purposes other than analysis and testing.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is illustrated by way of example, and not by way oflimitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements.

FIG. 1 illustrates a system that includes an analyzer and adevice-under-analysis, according to one embodiment.

FIG. 2 illustrates a system that includes an analyzer and a multi-devicesystem, the multi-device system including a device-under-analysis and adevice-under-observation, according to one embodiment.

FIG. 3 illustrates a flowchart of a method for testing and analyzing theimplementation quality of a device-under-analysis, according to oneembodiment.

DETAILED DESCRIPTION

As described above, a defectively designed or implemented device cannegatively affect the health of a system of which the device is amember. However, it is difficult to detect the negative effects in thesystem or to identify their causes. Therefore, what is needed is asystem and method that can test and analyze a device by a) determiningwhether invalid traffic would cause the device to negatively affect thesystem to which it belongs and b) identifying the causes of the negativeeffects.

In the following description, test and analysis for implementationquality (e.g., security, conformance, interoperability, robustness) of adevice are being performed. However, the description also applies toother types of device analysis and/or testing. “Device,”“device-under-analysis,” and “DUA” represent software and/or hardware.Software includes, for example, applications, operating systems, and/orcommunications systems (or subsystems). Hardware includes, for example,one or more devices. A device may be, for example, a mobile device(including a phone, personal digital assistant (PDA), or laptop),switch, bridge, router (including wireline or wireless), packet filter,firewall (including stateful or deep inspection), Virtual PrivateNetwork (VPN) concentrator, Network Address Translation (NAT)-enableddevice, proxy (including asymmetric), intrusion detection/preventionsystem, or network protocol analyzer.

System Architecture

FIG. 1 is a high-level block diagram of a system 100 for testing and/oranalyzing the implementation quality of a device-under-analysisaccording to one embodiment. The system 100 includes an analyzer 110 anda device-under-analysis (DUA) 120. The analyzer 110 and the DUA 120 arecommunicatively connected through a message channel 130 and a monitoringchannel 140. The message channel 130 and the monitoring channel 140 caneach use any type of data transmission mechanism, such as a network(wired or wireless). Examples of such a network include the Internet, anintranet, a wireless local area network (LAN), a cellular network, orany combination thereof. There may be devices located between theanalyzer 110 and the DUA 120 on the message channel 130 or themonitoring channel 140, although FIG. 1 omits such devices for clarity.

The analyzer 110 includes hardware and/or software devices that areconfigured to test and/or analyze the implementation quality of the DUA120. The DUA 120 includes hardware and/or software devices subject tothe test and/or analysis. The analyzer 110 can execute test cases totest the implementation quality of the DUA 120. A test case can includeone or more test messages that may expose an implementation flaw of theDUA 120. In one embodiment, the analyzer 110 generates and sends testmessages to the DUA 120 and receives responses from the DUA 120 throughthe message channel 130. Detailed information and examples about theanalyzer 110 and security tests for communication protocols may be foundin U.S. application Ser. No. 11/351,403, filed on Feb. 10, 2006, thecontent of which is incorporated by reference in its entirety. Theanalyzer 110 can monitor the DUA 120 to determine its operation. In oneembodiment, the DUA 120 outputs data that is sent to the analyzer 110using the monitoring channel 140. In another embodiment, the analyzer110 sends a command to the DUA 120 using the monitoring channel 140.

In one embodiment, the DUA 120 has capacity to execute a softwareapplication. For example, the DUA 120 can have a computer systemsupporting an embedded computing environment that can execute softwareapplications. The computer system can include one or more processors,memories, network interfaces, and display interfaces. Examples of thecomputing environment include a Java Virtual Machine (JVM) and anoperating system (e.g., Linux, Palm OS, Microsoft Windows Mobile). Inone embodiment, the DUA 120 is a mobile phone running a JVM.

In one embodiment, the analyzer 110 and the DUA 120 are structured toinclude a processor, memory, storage, network interfaces, and applicableoperating system and other functional software (e.g., network drivers,communication protocols).

Meta-Instrumentation

FIG. 2 is a high-level block diagram of a system 200 for testing and/oranalyzing the implementation quality of member devices in a multi-devicesystem according to one embodiment. Similar to the system 100, thesystem 200 also includes the analyzer 110 and the DUA 120, which arecommunicatively connected via the message channel 130. Unlike the system100, in the system 200 the DUA 120 is a part of a multi-device system210, which also includes a device-under-observation (DUO) 220.

The multi-device system 210 can include other member devices in additionto the DUA 120 and the DUO 220. The other member devices can havestructure and/or capacity similar to the DUA 120 and/or the DUO 220. Inone embodiment, any member device in the multi-device system 210 can bethe DUA 120 or the DUO 220. Member devices of the multi-device system210 may be alike (e.g., routers in a network) or dissimilar (e.g.firewalls, routers, web servers, etc.).

The DUO 220 includes hardware and/or software devices through which theanalyzer 110 can monitor the health of the multi-device system 210. Inone embodiment, similar to the DUA 120, the DUO 220 is structured toinclude a processor, memory, storage, network interfaces, and applicableoperating system and other functional software (e.g., network drivers,communication protocols), and has capacity to execute a softwareapplication.

The DUO 220 and the analyzer 110 are communicatively connected through asystem monitoring channel (or meta-instrumentation channel) 230. The DUA120 and the DUO 220 are communicatively connected through an impactchannel (or system channel) 240. Similar to the monitoring channel 140in FIG. 1, the system monitoring channel 230 and the impact channel 240can use any type of data transmission mechanism, such as a network(wired or wireless). The system monitoring channel 230, the impactchannel 240, and the message channel 130 could all be logicalconnections sharing the same physical communication medium or could eachhave a dedicated communication medium. There may be devices locatedbetween the analyzer 110 and the DUA 120 on the message channel 130,between the analyzer 110 and the DUO 220 on the system monitoringchannel 230, or between the DUA 120 and the DUO 220 over the impactchannel 240. In one embodiment, as further described below, the DUA 120is not located on the system monitoring channel 230.

The system monitoring channel 230 can be established using communicationprotocols (e.g., Ethernet, WiMAX, Wi-Fi, Bluetooth) and relatedparameters (e.g., security keys). These protocols and parameters can beidentified in a configuration stored in the analyzer 110. Alternatively,they may be automatically detected by the analyzer 110.

In one embodiment, the DUA 120 may affect operations of the DUO 220. Itis not necessary for the impact channel 240 to support directcommunications between the DUA 120 and the DUO 220. For example, the DUA120 may affect an intermediate device though the impact channel 240,which in turn affects the behavior of the DUO 220 through the impactchannel 240. Therefore, the DUA 120 may affect the DUO 220 through theimpact channel 240, even though the impact channel 240 does not supportdirect communications between the two.

The analyzer 110 can monitor the health of the multi-device system 210through the DUO 220. As described above, the DUA 120 may negativelyaffect the health of the multi-device system 210 when under attack. Thenegative effects may be visible to the DUO 220 or even affect theoperation of the DUO 220. For example, the DUA 120 may contaminate asystem-wide variable (or shared data structure) of the multi-devicesystem 210 (e.g., a routing table of a network). The DUO 220 can detectthis contamination by periodically checking the system-wide variables.The contaminated system-wide variable may also cause the DUO 220 tomalfunction. For example, a router may fail to deliver packets to theirdestinations if its routing table is contaminated.

The analyzer 110 can test and determine whether the DUA 120 maynegatively affect the multi-device system 210 by sending test cases tothe DUA 120 and monitoring the health of the multi-device system 210through the DUO 220. The process of testing the DUA 120 and monitoringthe health of the multi-device system 210 is calledmeta-instrumentation. Because the DUO 220 can detect impacts in themulti-device system 210 caused by the DUA 120 (hereinafter called“system-wide impacts of the DUA 120”), the analyzer 110 can detect thesystem-wide impacts of the DUA 120 caused by the test cases. Theanalyzer 110 can also transmit instrumentation vectors to the DUO 220 tomonitor its behavior changes during the test.

The analyzer 110 can monitor (or detect or observe) the ongoing healthof the multi-device system 210 through passive monitoring or activemonitoring. Passive monitoring includes reviewing information madeavailable by the DUO 220, while active monitoring includes executingcommands or function calls in the DUO 220 in order to obtain specificinformation. As an example of active monitoring, the value of asystem-wide variable can be checked by calling application programinterfaces (APIs) supported by the DUO 220. As an example of passivemonitoring, the outputs (e.g., logging file such as the syslog, outgoingcommunication such as console messages) of software applications (orconstituent or dependent process or thread) running on the DUO 220 canbe observed. In one embodiment, a monitor pattern feature is available.A monitor pattern is a regular expression designed to match keywords infault messages generated by the DUO 220. In this embodiment, a monitorpattern is used to identify the fault messages.

In one embodiment, the analyzer 110 can analyze a detected system-wideimpact of the DUA 120 to identify its cause. For example, the analyzer110 can establish a baseline snapshot of a system-wide variable of themulti-device system 210 and/or the DUO 220's state when the DUA 120 isoperating normally (e.g., before the analyzer 110 starts sending anytest case to the DUA 120). The baseline snapshot thus serves as ageneral mechanism to detect system-wide impacts of the DUA 120.Subsequently, snapshots of the system-wide variable can be obtainedperiodically during test cases. The monitoring activity can besynchronous or asynchronous with respect to the test cases. If a latersnapshot differs, it can be determined that at least one test case thatoccurred before that differing snapshot caused the system-wide impacts.As another example, if the DUO 220 malfunctions after a test case issent to the DUA 120, it can be determined that the test case causessystem-wide impacts, which in turn cause the DUO 220 to malfunction.

In one embodiment, the analyzer 110 can analyze the implementationquality of the DUA 120 based on the detected system-wide impacts of theDUA 120 and their causes. For example, the analyzer 110 can identifypotential implementation defects in the DUA 120 based on the test casescausing system-wide impacts. In one embodiment, the analyzer 110establishes a monitoring channel to the DUA 120 similar to themonitoring channel 140 illustrated in FIG. 1 to observe the DUA 120'sresponses to the test cases. In this embodiment, the analyzer 110 cananalyze the implementation quality of the DUA 120 based on its responsesto the test cases and the detected system-wide impacts.

In one embodiment, the analyzer 110 keeps a fault log and creates anentry in the log when it discovers a system-wide impact or a fault (orfault condition, internal failure) of a member device of themulti-device system 210 during an analysis. In one embodiment, an entrycontains various pieces of information, such as when the system-wideimpact or fault was discovered, which system-wide variable is affected,and which test message (or range or group of messages) caused thesystem-wide impact or fault.

In one embodiment, the analyzer 110 interacts with the DUO 220 using acommunication protocol different from the one the analyzer 110 uses tointeract with the DUA 120. In one embodiment, the analyzer 110 canmonitor system-wide impacts of the DUA 120 through the DUA 120 itselfusing a monitoring channel with a communication protocol and/or aphysical interface different from the one for sending test cases. Inanother embodiment, the analyzer 110 can monitor the health of themulti-device system 210 through multiple member devices.

In one embodiment, the analyzer 110, the DUA 120, and/or the DUO 220 canbe stored and operated on a single computer or on separate computersystems communicating with each other through a network.

Meta-Instrumentation Mechanism

Sometimes a test case can cause a DUA to negatively affect the health ofa system of which the DUA is a member device. If this happens, the DUAmay still respond to the test case and subsequent test cases properly,even though the negative effects may cause other member devices of thesystem to fail. In one embodiment, the analyzer 110 can test the DUAwhile monitoring the ongoing health of the system and detecting anynegative effect in the system caused by the DUA (e.g., through anothermember device of the system).

FIG. 3 is a flowchart of a method 300 for testing and analyzing theimplementation quality of a DUA 120 in a multi-device system 210 inaccordance with one embodiment. The method 300 can be implemented andperformed by the system 200. An example of the DUA 120 and themulti-device system 210 is a router and a network within which therouter resides.

The method 300 begins when a communication link (the message channel 130communication link) is established 310 between the analyzer 110 and theDUA 120 through the message channel 130. This establishment can beinitiated by the analyzer 110 or the DUA 120. For example, the analyzer110 can be configured with a network address of the DUA 120 andsupported communication protocols and can use this information toestablish 310 the message channel 130 communication link. Alternatively,the analyzer 110 can be configured to discover and connect to the DUA120 following a bootstrap/discovery procedure.

A communication link (the system monitoring channel 230 communicationlink) is established 320 between the analyzer 110 and the DUO 220through the system monitoring channel 230. A communication protocol ofthe system monitoring channel 230 communication link can be differentfrom a communication protocol of the message channel 130 communicationlink. The system monitoring channel 230 communication link can beestablished 320 before, simultaneously as, or after the message channel130 communication link is established 310.

In one embodiment, the system monitoring channel 230 communication linkdoes not pass through the DUA 120. The analyzer 110 puts the DUA 120under attack to determine whether it negatively affects the health ofthe multi-device system 210. As a result, the DUA 120 may not be stable.Therefore, the system monitoring channel 230 communication link, thechannel for the analyzer 110 to monitor the ongoing health of themulti-device system 210, should not depend on the DUA 120.

The analyzer 110 transmits (or sends) 330 tests cases to the DUA 120through the message channel 130 communication link. In one embodiment,the analyzer 110 can generate (or create) the test cases by generatingtest messages based on the information about the DUA 120 (e.g.,supported communication protocols, software and/or hardwareconfiguration). For example, the analyzer 110 can generate test casestargeted to test the communication protocols used to establish themessage channel 130 communication link. Alternatively, the analyzer 110can reuse existing test cases and transmit 330 them to the DUA 120.

In one embodiment (not shown), the analyzer 110 establishes a baselinesnapshot of system-wide variables of the multi-device system 210 and/ora state of the DUO 220 (or other member devices of the multi-devicesystem 210) before transmitting 330 the test cases to the DUA 120.

In one embodiment, only one member device of the multi-device system210, the DUA 120, is placed under attack at a time. The other memberdevices, because they are not under attack, are presumed to functionproperly. Therefore, if the analyzer 110 detects any change in themulti-device system 210 that negatively affects the health of themulti-device system 210 during the test cases, it can be determined thatthe DUA 120 caused the change.

The analyzer 110 monitors (or detects or observes) 340 the health of themulti-device system 210 through the system monitoring channel 230communication link. For example, the analyzer 110 can periodicallyestablish snapshots of the system-wide variables (e.g., data structuresshared among member devices). As another example, the analyzer 110 canmonitor 340 the operation or state of member devices of the multi-devicesystem 210 (e.g., the DUO 220).

As described above with respect to FIG. 2, the analyzer 110 can monitor340 actively and/or passively. Passive monitoring includes reviewinginformation made available by the DUO 220 and/or other member devices,while active monitoring includes executing commands or function calls inthe DUO 220 or other member devices in order to obtain specificinformation.

As an example of active monitoring, the analyzer 110 can periodically(e.g., once per second or once per test case) establish a snapshot ofsystem-wide variables by sending a query to the DUO 220 (or requestingthe DUO 220 to send the query to a data source hosting the system-widevariables) for the system-wide variables. The DUO 220 will respond byproviding information about the system-wide variables (e.g., currentvalue, last update time) to the analyzer 110. Alternatively, theanalyzer 110 can set a trigger in the DUO 220 (or another member device)for a system-wide variable, such that the DUO 220 will report to theanalyzer 110 when the system-wide variable is updated, or when anattempt to update the system-wide variable was made, or when a state ofthe DUO 220 is reached or changed. The trigger may also enable the DUO220 to periodically report information about its state and/or operationsto the analyzer 110.

As an example of passive monitoring, the analyzer 110 can passivelyobserve the outputs of the DUO 220 and match the observed outputs withmonitor patterns to identify messages about events of interest, such asoperation failure, internal state change, etc. For example, when testinga multi-device system comprising routers in a network supporting theOSPF routing protocol, the analyzer 110 can passively observe the output(and/or input) of a router for messages updating the LSDB.

In one embodiment, the analyzer 110 monitors 340 the operation or stateof the DUO 220 (or other member devices of the multi-device system 210)by sending instrumentation vectors to the DUO 220 and observing itsresponses to the instrumentation vectors. The goal of theinstrumentation vectors is for the analyzer 110 to monitor operations ofthe DUO 220 and identify any faults (or fault conditions, internalfailures) in the DUO 220. The instrumentation vectors sent to the DUO220 may be of the same communication protocol as the test casestransmitted 330 to the DUA 120, or of a different communicationprotocol.

In one embodiment, the instrumentation vectors can be of multiplecommunication protocols. For example, in a network supporting the OSPFrouting protocol and the Multiprotocol Label Switching (MPLS) switchingprotocol, invalid MPLS packets may cause topology changes that mayaffect routers supporting the OSPF routing protocol. Therefore, theanalyzer 110 can send test messages to a first router (the DUA 120)using the MPLS switching protocol, and use an OSPF connection to asecond router (the DUO 220) to observe whether the LSDB or the routingtable is changed by the test messages sent to the first router againstits MPLS layer. If the LSDB or the routing table changes, the analyzer110 can determine that the particular test messages caused incorrectbehavior in the first router.

In one embodiment, the analyzer 110 monitors 340 the operation or stateof the DUO 220 through an internal agent residing inside the DUO 220.The internal agent can monitor the internal state changes and operationsof the DUO 220 caused by the DUA 120 responding to the test cases.Detailed information and examples about the internal agent may be foundin U.S. application Ser. No. 11/696,605, filed on Apr. 4, 2007, thecontent of which is incorporated by reference in its entirety.

The analyzer 110 analyzes 350 the implementation quality of the DUA 120based on information it monitored through the system monitoring channel230 communication link. For example, the analyzer 110 can execute afault isolation algorithm to identify particular test cases transmittedto the DUA 120 that negatively affect the health of the multi-devicesystem 210. For example, the analyzer 110 can compare snapshots takenduring test cases to identify changes. The analyzer 110 determineswhether the identified changes in the snapshots are valid. For example,if the routing table of a network is changed and there is no topologychange for the network, it can be determined that the change is invalid.If the analyzer 110 determines that invalid changes have happened, itcan infer that the test cases caused incorrect behavior in the DUA 120.

The analyzer 110 can analyze 350 the implementation quality of the DUA120 using various information (e.g., information obtained through activeor passive monitoring). In one embodiment, the analyzer 110 can monitorthe state and/or operation of the DUA 120 by establishing a monitoringchannel to the DUA 120 similar to the monitoring channel 140 in FIG. 1,and conduct analysis based on this information. The analyzer 110 may beconfigured to conduct analysis on a fixed interval schedule or conductanalysis at the end of each test case or each set of test cases.

Based on the result of the analysis, the analyzer 110 can transmit 330more test cases to the DUA 120 (or another member device of themulti-device system 210), conduct further analysis 350, or generate areport summarizing its findings.

The embodiments described herein beneficially use a first member deviceof a system to observe the health of the system while testing a secondmember device. Therefore, the embodiments can conduct output validationof the second member device by determining whether it negativelyaffected the system when under test. The embodiments can also identifytest case(s) causing incorrect behavior in the member devices.

This disclosed system and method can be applied to a wide field ofdevices (including wireless devices and battery-powered devices) andusing various types of testing and/or analysis (e.g., security,conformance, interoperability, robustness).

In the preceding description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the disclosure. It will be apparent, however, to oneskilled in the art that the disclosure can be practiced without thesespecific details. In other instances, structures and devices are shownin block diagram form in order to avoid obscuring the disclosure.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment.

Some portions of the detailed descriptions that follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, objects, symbols,characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission, or display devices.

The present disclosure also relates to an apparatus for performing theoperations herein. This apparatus is specially constructed for therequired purposes, or it comprises a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program is stored in a computer readablestorage medium, such as, but not limited to, any type of disk includingfloppy disks, optical disks, CD-ROMs, and magnetic-optical disks,read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, and each coupled to a computer systembus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems are used with programs in accordance with the teachings herein,or more specialized apparatus are constructed to perform the requiredmethod steps. The required structure for a variety of these systemsappears in the description herein. In addition, the present disclosureis not described with reference to any particular programming language.It will be appreciated that a variety of programming languages may beused to implement the teachings of the disclosure as described herein.

1. A method for testing and analyzing a security vulnerability of amulti-device network system to protocol abuse of a networkcommunications protocol, the method implemented by a security analyzerdevice, comprising: establishing a first communication link between amember network device-under-analysis (DUA) of the multi-device networksystem and the security analyzer device; establishing a secondcommunication link between a member network device-under-observation(DUO) of the multi-device network system and the security analyzerdevice, the DUA and the DUO being distinct member devices of themulti-device network system; attacking the DUA multiple times, theattacks comprising sending to the DUA through the first communicationlink test messages that are invalid with respect to the networkcommunication protocol; during and after attacking the DUA, monitoring,with the security analyzer, outputs from the DUO received over thesecond communication link; determining, based on the outputs receivedfrom the DUO, whether the multi-device network system includes asecurity vulnerability; responsive to a determination that themulti-device network system includes a security vulnerability, based onthe outputs received from the DUO, identifying which attack causes thesecurity vulnerability.
 2. The method of claim 1, wherein the securityanalyzer device is not a member device of the multi-device networksystem, and wherein determining whether the multi-device network systemincludes a security vulnerability comprises (1) determining whether theattacks cause a system-wide impact through the DUA in the multi-devicenetwork system and (2) determining whether the system-wide impactcomprises a security vulnerability.
 3. The method of claim 2, whereinthe system-wide impact comprises a change or an attempt to change asystem-wide variable.
 4. The method of claim 3, wherein the system-widevariable comprises at least one selected from a group consisting of arouting table and a shared database.
 5. The method of claim 2, whereinthe system-wide impact comprises a malfunction in a member networkdevice of the multi-device network system.
 6. The method of claim 1,wherein a communication protocol of the first communication link isdifferent from a communication protocol of the second communicationlink.
 7. The method of claim 1, further comprising: monitoring responsesof the DUA to the attacks, wherein determining whether the multi-devicenetwork system includes a security vulnerability further comprisesanalyzing the DUA based on the responses.
 8. The method of claim 1,further comprising: generating the attacks based on one of the followinginformation: a supported communication protocol of the DUA, a softwareconfiguration of the DUA, and a hardware configuration of the DUA. 9.The method of claim 1, wherein the second communication link does notpass through the DUA.
 10. A computer program product for testing andanalyzing a security vulnerability of a multi-device network system toprotocol abuse of a network communications protocol, the methodimplemented by a security analyzer device, the computer program productcomprising a computer-readable medium containing computer program codefor performing a method comprising: establishing a first communicationlink between a member network device-under-analysis (DUA) of themulti-device network system and the security analyzer device;establishing a second communication link between a member networkdevice-under-observation (DUO) of the multi-device network system andthe security analyzer device, the DUA and the DUO being distinct memberdevices of the multi-device network system; attacking the DUA multipletimes, the attacks comprising sending to the DUA through the firstcommunication link test messages that are invalid with respect to thenetwork communication protocol; during and after attacking the DUA,monitoring, with the security analyzer, outputs from the DUO receivedover the second communication link; determining, based on the outputsreceived from the DUO, whether the multi-device network system includesa security vulnerability; responsive to a determination that themulti-device network system includes a security vulnerability, based onthe outputs received from the DUO, identifying which attack causes thesecurity vulnerability.
 11. The computer program product of claim 10,wherein the security analyzer device is not a member device of themulti-device network system, and wherein determining whether themulti-device network system includes a security vulnerability comprises(1) determining whether the attacks cause a system-wide impact throughthe DUA in the multi-device network system and (2) determining whetherthe system-wide impact comprises a security vulnerability.
 12. Thecomputer program product of claim 11, wherein the system-wide impactcomprises a change or an attempt to change a system-wide variable. 13.The computer program product of claim 11, wherein the system-wide impactcomprises a malfunction in a member network device of the multi-devicenetwork system.
 14. The computer program product of claim 10, whereinthe system-wide variable comprises at least one selected from a groupconsisting of a routing table and a shared database.
 15. The computerprogram product of claim 10, wherein a communication protocol of thefirst communication link is different from a communication protocol ofthe second communication link.
 16. The computer program product of claim10, wherein the method further comprises: monitoring responses of theDUA to the attacks, wherein determining whether the multi-device networksystem includes a security vulnerability further comprises analyzing theDUA based on the responses.
 17. The computer program product of claim10, wherein the method further comprises: generating the attacks basedon one of the following information: a supported communication protocolof the DUA, a software configuration of the DUA, and a hardwareconfiguration of the DUA.
 18. The computer program product of claim 10,wherein the second communication link does not pass through the DUA. 19.A security analyzer device for testing and analyzing a securityvulnerability of a multi-device network system to protocol abuse of anetwork communications protocol, comprising: a computer processorconfigured to execute computer program instructions; and acomputer-readable storage medium having executable computer programinstructions tangibly embodied thereon, the executable computer programconfigured to cause the computer processor to perform the steps of:establishing a first communication link between the DUA and an analyzer,wherein the analyzer is external to the DUA; establishing a secondcommunication link between a device-under-observation (DUO) and theanalyzer, the DUA and the DUO being distinct member devices of amulti-device system; transmitting to the DUA via the first communicationlink a test case including a message which is malformed or invalid withrespect to the network communications protocol; monitoring via thesecond communication link a system-wide impact of the DUA in themulti-device system caused by the test case; and analyzing whether theDUA has a security vulnerability based on the test case and thesystem-wide impact.
 20. A security analyzer device for testing andanalyzing a security vulnerability of a multi-device network system toprotocol abuse of a network communications protocol, comprising: acomputer processor configured to execute computer program instructions;and a computer-readable storage medium having executable computerprogram instructions tangibly embodied thereon, the executable computerprogram configured to cause the computer processor to perform the stepsof: establishing a first communication link between the DUA and ananalyzer, wherein the analyzer is external to the DUA; establishing asecond communication link between a device-under-observation (DUO) andthe analyzer, the DUA and the DUO being distinct member devices of amulti-device system; transmitting to the DUA via the first communicationlink a test case; monitoring via the second communication link asystem-wide impact of the DUA in the multi-device system caused by thetest case; and analyzing the DUA based on the test case and thesystem-wide impact, wherein the security analyzer device is not a memberdevice of the multi-device network system, and wherein determiningwhether the multi-device network system includes a securityvulnerability comprises (1) determining whether test case causes asystem-wide impact through the DUA in the multi-device network systemand (2) determining whether the system-wide impact comprises a securityvulnerability.